System and method for finding and classifying lines in an image with a vision system

ABSTRACT

This invention provides a system and method for finding line features in an image that allows multiple lines to be efficiently and accurately identified and characterized. When lines are identified, the user can train the system to associate predetermined (e.g. text) labels with respect to such lines. These labels can be used to define neural net classifiers. The neural net operates at runtime to identify and score lines in a runtime image that are found using a line-finding process. The found lines can be displayed to the user with labels and an associated probability score map based upon the neural net results. Lines that are not labeled are generally deemed to have a low score, and are either not flagged by the interface, or identified as not relevant.

RELATED APPLICATION

This application is a continuation-in-part of co-pending U.S. patentapplication Ser. No. 15/338,445, entitled SYSTEM AND METHOD FOR FINDINGLINES IN AN IMAGE WITH A VISION SYSTEM, filed Oct. 31, 2016, whichclaims the benefit of U.S. Provisional Application Ser. No. 62/249,918,entitled SYSTEM AND METHOD FOR FINDING LINES IN AN IMAGE WITH A VISIONSYSTEM, filed Nov. 2, 2015, the teachings of each of which applicationsis incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates to machine vision systems, and more particularlyto vision system tools that find line features in acquired images

BACKGROUND OF THE INVENTION

Machine vision systems (also termed herein, simply “vision systems”) areused for a variety of tasks in manufacturing, logistics, and industry.Such tasks can include surface and part inspection, alignment of objectsduring assembly, reading of patterns and ID codes, and any otheroperation in which visual data is acquired and interpreted for use infurther processes. Vision systems typically employ one or more camerasthat acquire images of a scene containing an object or subject ofinterest. The object/subject can be stationary or in relative motion.Motion can also be controlled by information derived by the visionsystem, as in the case of manipulation of parts by a robot.

A common task for a vision system is finding and characterizing linefeatures in an image. A variety of tools are used to identify andanalyze such line features. Typically, these tools rely upon a sharpcontrast difference that occurs in a portion of the image. This contrastdifference is analyzed using e.g. a caliper tool to determine if theindividual points in the image with contrast difference can be assembledinto a line-like feature. If so, then a line is identified in the image.Notably, the tools that find edge points and those that attempt to fit aline to the points act independently of each other. This increasesprocessing overhead and decreases reliability. Where an image containsmultiple lines, such tools may be limited in ability to accuratelyidentify them. Furthermore, traditional, line-finding tools that aredesigned to find a single line in an image can be problematic to usewhen the image contains multiple closely spaced lines with similarorientation and polarity.

Another challenge is that lines of an object may sometimes be occludedor unclear in an acquired image. The user may be unsure as to theidentity of found lines, and mechanisms that allow discreteidentification of such, can involve the writing of sophisticated rulesand scripts, which adds time and cost to a vision system setup andtraining task.

SUMMARY OF THE INVENTION

This invention overcomes disadvantages of the prior art by providing asystem and method for finding line features in an image that allowsmultiple lines to be efficiently and accurately identified andcharacterized. When lines are identified, the user can train the systemto associate predetermined (e.g. text) labels with respect to suchlines. These labels (also termed herein “tags”) can be used to defineneural net classifiers. The neural net operates at runtime to identifyand score lines in a runtime image that are found using a line-findingprocess. The found lines can be displayed to the user with labels and anassociated probability score map based upon the neural net results.Lines that are not labeled are generally deemed to have a low score, andare either not flagged by the interface, or identified as not relevant.

In an illustrative embodiment, a system and method for finding linefeatures in an acquired image, based upon images acquired by one or morecameras, is provided. The system and method includes a vision systemprocessor, and an interface associated with the vision system processor,that allows creation of discrete labels with respect to relevant lineslocated by a line-finding process in a training image of the object. Aruntime line-finding process locates lines in an acquired image and aneural net process employs one or more classifier(s), based on thelabels, to determine a probability map for line features relative to thelabels. A runtime result-generation process provides labels andprobability scores for at least one of the relevant lines.Illustratively, the runtime result-generation process providesprobability scores for non-relevant lines, and/or includes an interfacethat highlights lines and provides the probability scores associatedwith the highlighted lines. The probably score map can be similar insize to the acquired image. The line-finding process can comprise aprocessor that receives image data of a scene containing line features,having an edge point extractor that (a) computes a gradient vector fieldfrom the image data, (b) projects the gradient vector field over aplurality of gradient projection sub-regions, and (c) finds a pluralityof edge points in respective of the gradient projection sub-regionsbased on the projected gradient data. The processor also comprises aline-finder that generates a plurality of lines that are consistent withthe edge points extracted from the image. Illustratively, theline-finder operates a RANSAC-based process to fit inlier edge points tonew lines including iteratively defines lines from outlier edge pointswith respect to previously defined lines. The gradient field projectioncan be oriented along a direction set in response to an expectedorientation of one or more or the line features and/or can define agranularity based on a Gaussian kernel. The edge point extractor can bearranged to find a plurality of gradient magnitude maxima in each of thegradient projection sub-regions. The gradient magnitude maxima arerespectively identified as some of the plurality edge points, beingdescribed by a position vector and a gradient vector. The line-findercan also be arranged to determine consistency between at least one edgepoint of the extracted plurality of edge points and at least onecandidate line of the found plurality of lines by computing a metricthat is based upon a distance of the at least one edge point from the atleast one candidate line and an angle difference between a gradientdirection of the at least one edge point and a normal direction of theat least one candidate line. Illustratively, the image data includesdata from a plurality of images acquired from a plurality of cameras.The images are thereby transformed into a common coordinate space.

In an illustrative embodiment, a system for finding line features in anacquired image based upon one or more cameras is provided. The systemand method includes a vision system processor and an interfaceassociated with the vision system processor, which allows creation ofdiscrete labels with respect to relevant lines located by a line-findingprocess in a training image of the object. A runtime line-findingprocess locates lines in an acquired image, and a statistical classifieror a K-NN classifier that produces the labels for the interface basedupon lines located by the line-finding process.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a diagram of an exemplary vision system arrangement acquiringimages of an object that includes multiple edge features and a visionsystem processor including an edge-finding tool/module and associatedlabel interface processor/module in accordance with an illustrativeembodiment;

FIG. 2 is a diagram showing an overview of the system and method foredge-point extraction and line-finding from an acquired image accordingto an illustrative embodiment;

FIG. 3 is a flow diagram of an edge point extraction procedure accordingto the system and method of FIG. 2;

FIG. 4 is a diagram of a field projection on a region of an imagecontaining edge features, which is part of the edge point extractionprocedure of FIG. 3;

FIG. 5 is a diagram showing application of a Gaussian kernel to theimage to smooth the image, for use in the edge point extractionprocedure of FIG. 3;

FIG. 6 is a diagram of a field projection, including application of aGaussian kernel for smoothing of the projection, for use in the edgepoint extraction procedure of FIG. 3;

FIG. 7 is a diagram showing a graphical overview of the edge pointextraction procedure of FIG. 3 including application of Gaussian kernelsand calculation of absolute and normalized contrast thresholds for edgepoints;

FIG. 8 is graph showing a region of qualified contrasts for edge pointshaving a sufficient absolute contrast threshold and normalized contrastthreshold;

FIG. 9 is a flow diagram of a line-finding procedure based upon foundedge points in FIG. 3, using an exemplary RANSAC process according to anillustrative embodiment;

FIGS. 10 and 11 are diagrams showing erroneous and correct alignment ofedge points relative to closely spaced parallel line features,respectively;

FIGS. 12 and 13 are diagrams showing correct and erroneous alignment ofedge points relative to crossing line features, respectively, which canbe resolved according to the line-finder of the illustrative system andmethod;

FIGS. 14-17 are diagrams showing examples of lines exhibiting,respectively, light-to-dark polarity, dark-to-light-polarity, eitherlight-to-dark or dark-to-light polarity, or mixed polarity, which can beresolved according to the line-finder of the illustrative system andmethod;

FIG. 18 is a diagram showing modification of the coverage score for afound line in view of a user-defined mask;

FIG. 19 is a flow diagram showing a procedure for training a line-findertool using an interface that includes labels/tags referring to lines ofinterest in an object image;

FIG. 20 is a diagram showing an exemplary user interface display,including dialog boxes and an image of a training object having lines ofinterest, used for performing the training procedure of FIG. 19;

FIG. 21 is a flow diagram of a runtime procedure for finding lines usingthe trained line-finder in accordance with FIG. 19, including theassigning of labels to lines of interest and (e.g. neural) netclassifier-based probability scores to found lines;

FIG. 22 is a diagram showing an exemplary user interface display,including an image of an object having found lines based upon theline-finder runtime procedure according to FIG. 21, showing a label andprobability score for a non-relevant line; and

FIG. 23 is a diagram showing an exemplary user interface display,including an image of an object having found lines based upon theline-finder runtime procedure according to FIG. 21, showing a label andprobability score for a line of interest.

DETAILED DESCRIPTION I. System Overview

An exemplary vision system arrangement 100 that can be employedaccording to an illustrative embodiment is shown in FIG. 1. The system100 includes at least one vision system camera 110, and can include oneor more additional, optional cameras 112 (shown in phantom). Theillustrative camera(s) 110, 112 include(s) an image sensor (or imager) Sand associated electronics for acquiring and transmitting image framesto a vision system process(or) 130 that can be instantiated in astandalone processor and/or a computing device 140. The camera 110 (and112) includes an appropriate lens/optics 116 focused upon a scene thatcontains an object 150 under inspection. The camera 110 (and 112) caninclude internal and/or external illuminators (not shown) that operatein accordance with the image acquisition process. The computing device140 can be any acceptable processor-based system capable of storing andmanipulating image data in accordance with the illustrative embodiment.For example, the computing device 140 can comprise a PC (as shown),server, laptop, tablet, smartphone or other similar device. Thecomputing device 140 can include appropriate peripherals, such as abus-based image capture card that interconnects to the camera. Inalternate embodiments, the vision processor can be partially or fullycontained within the camera body itself and can be networked with otherPCs, servers and/or camera-based processors that share and process imagedata. The computing device 140 optionally includes an appropriatedisplay 142, which can support an appropriate graphical user interface(GUI) that can operate in accordance with vision system tools andprocessors 132 provided in the vision system process(or) 130. Note thata display can be omitted in various embodiments and/or provided only forsetup and service functions. The vision system tools can be part of anyacceptable software and/or hardware package that is acceptable for usein the inspection of objects, such as those commercially available fromCognex Corporation of Natick, Mass. The computing device can alsoinclude associated user interface (UI) components, including, forexample, a keyboard 144 and mouse 146, as well as a touchscreen withinthe display 142.

The camera(s) 110 (and 112) image some or all of an object 150 locatedwithin the scene. Each camera defines an optical axis OA, around which afield of view is established based upon the optics 116, focal distance,etc. The object 150 includes a plurality of edges 152, 154 and 156 thatare respectively arranged in different directions. For example, theobject edges can comprise those of a cover glass mounted within asmartphone body. Illustratively, the camera(s) can image the entireobject, or specific locations (e.g. corners where the glass meets thebody). A (common) coordinate space can be established with respect tothe object, one of the cameras or another reference point (for example amoving stage upon which the object 150 is supported). As shown, thecoordinate space is represented by axes 158. These axes illustrativelydefine orthogonal x, y and z axes and rotation Oz about the z axis inthe x-y plane.

According to an illustrative embodiment, the vision system process 130interoperates with one or more applications/processes (running on thecomputing device 140) that collectively comprise a set of vision systemtools/processes 132. These tools can include a variety of conventionaland specialized applications that are used to resolve image data—forexample a variety of calibration tools and affine transform tools can beused to transform acquired image data to a predetermined (e.g. common)coordinate system. Tools that convert image grayscale intensity data toa binary image based upon a predetermined threshold can also beincluded. Likewise, tools that analyze the gradient of intensity(contrast) between adjacent image pixels (and subpixels) can beprovided.

The vision system process(or) 130 includes a line-finding process, toolor module 134 that locates multiple lines in an acquired image accordingto an illustrative embodiment. Reference is, thus, made to FIG. 2, whichgraphically depicts an overview of a line-finding procedure 200according to the illustrative embodiment. The procedure 200 consists oftwo primary parts. An input image 210 is provided to the processor. Asshown, the image includes a pair of intersecting edges 212 and 214.These can represent a corner region of the above-described object 150.An edge point extractor 220 processes the input image 210 to obtain aset 230 of candidate edge points, including edge points 232 and 234 thatrespectively reside along edges 212 and 214. The edge points 232, 234,and their associated data (e.g. intensity gradient information describedbelow), are provided to a recursive line-finder 240, which performs aseries of iterative processes on selected edge points. The goal of theiterative processes is an attempt to fit other found edge points tocandidate line features. The line-finding process 240 results in foundlines 252 and 254 as shown. These results can be provided to otherdownstream processes 260 that use the information—e.g. alignmentprocesses, robot manipulation, inspection, ID reading, part/surfaceinspection, etc.

II. Line-Finding Process(or)

Reference is made to FIG. 3, which describes a procedure for extractingedge points according to an embodiment. One or more images are acquiredof the scene, which contains an object or surface having edge featuresto be found (step 310). The image(s) can be extracted by a single cameraor by a plurality of cameras. In either case, the image pixels can be(optionally) transformed by appropriate calibration parameters to a newand/or common coordinate space in step 320. This step can also includesmoothing of the image as described below. In certain embodiments, wherea plurality of cameras image discontinuous regions of the scene—forexample focusing on corner regions of a larger object—the commoncoordinate space can account for the empty region between camera fieldsof view. As described below, lines that extend between such fields ofview (e.g. the object edge that connects two found corner regions) canbe extrapolated by the system and method of the illustrative embodiment.The edge points required for finding lines are extracted from theimage(s) in the appropriate coordinate space by the edge point extractorusing gradient field projection in step 330. Gradient values are firstcomputed for each pixel, generating two images for x and y gradientcomponents. The image(s) are further processed by projecting thegradient field over many caliper-like regions. Unlike a conventionalcaliper tool which projects the intensity value, by projecting thegradient field in accordance with the embodiment, gradient orientationcan be preserved, which facilitates the subsequent line-finding processas described below.

In step 340, and also referring to the diagram of FIG. 4, a portion (acaliper-like region) 400 of the image containing the candidate edgefeatures is subjected to a gradient field projection (represented by aplurality of projections 410, 420, 430, searched across the(approximately) expected orientation of the edges in a search direction(arrow SD), with the projections repeated across the region 400 in anorthogonal projection direction (arrow PD). For each projection (e.g.projection 420) edges appear as local maxima in a gradient field 440associated with the projection. In general, a series of edge pointswithin the projection that are associated with an edge will exhibit anintensity gradient (vectors 552, 554) orthogonal to the direction ofextension of the edge. As described below, the user can define theprojection direction based on expected line orientation. Alternatively,this can be provided by default or by another mechanism—e.g. analysis ofthe features in the image.

Two granularity parameters are involved in the above-described gradientprojection step. Prior to gradient field calculation, the user canchoose to smooth the image using an isotropic Gaussian kernel. A firstgranularity determines the size of this Gaussian smoothing kernel. Asshown in the diagram 500 of FIG. 5, an application of an appropriatelysized Gaussian kernel (e.g. large 512, medium 514, small 516) is made tosmooth the image 210. The first granularity parameter, hence, determinesthe size of the isotropic Gaussian smoothing kernel prior to fieldcalculation.

After gradient field calculation, a Gaussian-weighted projection isthereby performed by the process, rather than uniform weighting inconventional caliper tools. Thus, a second granularity parameterdetermines the size of the one-dimensional (1D) Gaussian kernel usedduring field projection as shown in FIG. 6, in which the region 600 issubjected to a Gaussian-smoothed kernel 610, 620, 630. During a typicaloperation, the user verifies (using the GUI) all extracted edgesoverlaid on the image, and then adjusts the granularities and contrastthresholds until the number of extracted edges along the lines to befound appears satisfactory, while avoiding an excessive number of edgesdue to background noise in the image. In other words, this step allowsthe signal-to-noise ratio to be optimized for the image characteristic.This adjustment can also be performed automatically by the system, usinga default value in various embodiments. Note that the use of a Gaussianweighting function is one of a variety of approaches for weighting theprojection, including (e.g.) a uniform weighting.

The overall flow of gradient field extraction and projection isillustrated graphically in the diagram 700 of FIG. 7. The twogranularity parameters, the isotropic Gaussian kernel 710 and the 1DGaussian kernel 720, are each shown in each half of the overall diagram700. As shown, each acquired image 210 is subjected to smoothing anddecimation 730. The resulting image 740 is then subjected to gradientfield calculation 750, as described above, to generate the two gradientimages 752 and 754. These gradient images are also represented as g_(x)and g_(y), representing two orthogonal axes in the common coordinatespace. Note that in addition to two gradient images, the intensity image756 is also typically subjected to the smoothing, decimation andprojection process 760 (using a Gaussian-weighted projection 770 basedupon the 1D Gaussian kernel 720) since the processed intensityinformation is also employed for calculating normalized contrasts inaccordance with an embodiment—described below. The result is theprojection profiles of gradient images 772 (g _(x)), 774 (g _(y)), andintensity image 776.

Referring also to step 350 of the procedure 300 (FIG. 3), qualified edgepoints are then extracted by combining the 1D projection profiles ofboth x & y gradient images. This is accomplished using a raw contrastcalculation 780 and a normalized contrast calculation 790 based on theIntensity image. More particularly, any local peaks with both rawprojected gradient magnitudes and normalized projected gradientmagnitudes exceeding respective thresholds are considered a candidateedge point for subsequent line-finding according to the followingillustrative equations:(g _(x) ² +g _(y) ²)^(1/2) >T _(ABS)(g _(x) ² +g _(y) ²)^(1/2) /I>T _(NORM)where g_(x) and g_(y) are the values of the x-gradient and y-gradientprojections at a pixel location, respectively, I the intensity, T_(ABS)an absolute contrast threshold for raw projected gradient magnitudes andT_(NORM) is a normalized contrast threshold for intensity-normalizedprojected gradient magnitudes.

Notably, a point is only considered a candidate edge point when itsabsolute and normalized contrasts both exceed their respectivethresholds. This is shown by the upper right quadrant 810 in theexemplary graph 800 of normalized contrast threshold T_(NORM) versusabsolute contrast threshold T_(ABS). The use of dual (absolute andnormalized) thresholds differs generally from existing approaches thattypically employ an absolute contrast threshold. The benefits of dualcontrast thresholds are clear, by way of example, when an image includesboth dark and bright intensity regions that both include edges ofinterest. In order to detect edges in dark regions of the image, it isdesirable to set a low contrast threshold. However, such a low contrastsetting can result in the detection of false edges in the brightportions of the image. Conversely, in order to avoid the detection offalse edges in the bright regions of the image, it is desirable to set ahigh contrast threshold. However, with a high contrast setting, thesystem may fail to adequately detect edges in dark regions of the image.By using a second normalized contrast threshold, in addition to thetraditional absolute contrast threshold, the system can appropriatelydetect edges both in dark and bright regions, and avoid detecting falseedges in bright regions of the image. Hence, by enabling the detectionof relevant edges while avoiding spurious edges, the use of dualcontrast thresholds serves to maximize the speed and robustness of thesubsequent line-finding stage of the overall process.

Referring further to procedure step 350 (FIG. 3), once all edge pointsare extracted, they are represented and stored in a data structure thatis convenient for subsequent line-finders to operate upon. Note, forexample, the following tuple:p=(x,y,gx,gy,gm,go,I,gm/I,m,n)where (x,y) is the location of the edge point, (g_(x),g_(y)) are thevalues of its respective x-gradient and y-gradient projections,(g_(m),g_(o)) is the gradient magnitude and orientation computed from(g_(x),g_(y)), I is the intensity at the edge point location, g_(m)/I isthe intensity-normalized contrast obtained by dividing the gradientmagnitude g_(m) by the intensity I, m is the image index and n is theprojection region index. The location of the edge point, as in thestandard caliper tool, can be interpolated for improved accuracy.

Note that the edge-point extraction process generally operates to runfield projections in a single direction that substantially matches theexpected line angle. The tool is, therefore, most sensitive to edges atthis angle, and its sensitivity falls off gradually for edges at otherangles, where the rate of fall-off depend on the granularity settingsthat indirectly determine the field projection length. As a result, theprocess is limited to finding lines whose angle is “near” the expectedline angle, subject to the angle range specified by the user. While theprocess is adapted to find lines that are not orthogonal, it iscontemplated that it could be generalized in various embodiments to findlines of any angle over 360 degrees by performing projections inmultiple directions (omnidirectional line-finding), including orthogonaldirections.

With reference now to step 360 of the procedure 300 (FIG. 3),thresholded edge point candidates are provided to the line-finder inaccordance with an illustrative embodiment. By way of example, theline-finder operates recursively and employs (e.g.) RANdom SAmpleConcensus (RANSAC)-based techniques. Reference is also made to theline-finding procedure 900 in FIG. 9. In step 910, the user specifiesthe maximum number of expected lines in an image, along with an expectedangle, angle tolerance, distance tolerance, and (illustratively) aminimum coverage score (defined generally below) via (e.g.) the GUI.These parameters are used by the line-finder to operate the followingprocesses. The lines are found for each subregion of the image byrecursively running a RANSAC line-finder, the edge point outliers fromone stage becoming the input points to the next stage. Thus, in step920, the procedure 900 selects a pair of edge points that are part ofthe group of edge points identified as extrema in the edge-findingprocess. The procedure 900 attempts to fit a model line to the selectededge points based on matching gradient values (within the selected rangeof tolerance) that are consistent with a model line. In step 924, one ormore line candidate(s) from step 922 is/are returned. Each line-findingstage returns a candidate line, its inliers and outliers. The returnedline(s) is/are subjected to a computation of inlier edge points thathave a position and gradient consistent with the line candidate (step926). In step 928, the candidate line with the greatest inlier count isidentified. The above-described line-finding stage (steps 920-928)terminates when it reaches the maximum number of RANSAC iterationsallowed (decision step 930). The maximum number of iterations insideeach line-finding stage are computed automatically using an internallycomputed worst case proportion of outliers and an assurance levelspecified by the user. Each line-finding stage returns the line with themaximum number of captured edge points out of all its iterations—subjectto a user-specified fit tolerance, geometric constraints and polarity.Each edge point can only be assigned to the inlier list of a single lineand each line is only allowed to contain at most one edge point fromeach projection region. The gradient orientation of an edge point, alongwith its position, is used to determine whether it should be included inthe inlier list of a candidate line. In particular, edge points shouldhave gradient orientation that is consistent with the angle of acandidate line.

If the decision step 930 determines that more iterations are permitted,the outliers from the best inlier candidate are returned (step 940) tothe RANSAC process (step 920) for use in finding a line candidate.

During each RANSAC iteration, two edge points belonging to differentprojection regions are randomly selected and a line will be fit to thosetwo points. The resulting candidate line receives further considerationonly if its angle is consistent with the gradient angles of both edgesin the point pair and if the angle of the line is consistent with theuncertainty range specified by the user. In general, the gradientdirection of an edge point is nominally orthogonal, but is allowed todiffer by a user-configured angle tolerance. If a candidate line passesthese initial tests, then the number of inlier edge points will beevaluated, otherwise a new RANSAC iteration is initiated. An edge pointwill be regarded as an inlier of a candidate line only if its gradientdirection and position are consistent with the line—based on gradientangle and distance tolerances specified by the user.

When the RANSAC iterations reach the maximum (decision step 930), theinliers of the best found line candidate are subjected to an improvedline fit, using (for example) a least squares regression or anotheracceptable approximation technique, and the set of inlier edge pointswill be reevaluated, repeating these steps a maximum of N (e.g. three ormore) times until the number of inliers ceases to further increase ordecrease (step 960). This is the line that is indicated as found in step970.

The decision step 980 determines whether more lines are to be found(based (e.g.) on searching further sub regions or another criteria), andif so, the process loops back to step 920 to operate on a new set ofedge points (step 982). When the points have been exhausted or a maximumiteration count is reached, the procedure 900 returns a set of (i.e.multiple) found lines in the image in step 990.

The multi-line-finder is adapted to perform a final adjustment ofexisting results in cases where two line results intersect one anotherwithin the inspection region. As illustrated generally in FIGS. 10 and11, for closely spaced parallel lines 1010 and 1020, erroneous lineresults (i.e. FIG. 10) can sometimes be obtained due to the statisticalnature of the RANSAC procedure. However, when such errors occur, anexchange of inlier point groups (arrow 1120 in groups 1110 in FIG. 11)can sometimes locate the correct lines with increased coverage scoresand reduced-fit residuals. Point exchanges can be most effective when animage contains closely spaced parallel lines as shown. Conversely, whenthe image contains lines 1210 and 1220 that actually do cross each otheras shown in FIGS. 12 and 13, then coverage scores are reduced after apoint exchange (arrow 1230 in group 1240 in FIG. 12), so the originalresults obtained before the exchange are retained by the process tosuccessfully detect crossing lines.

Note that the RANSAC procedure is one of a variety of techniques bywhich the line-finder can fit points to a line. In alternateembodiments, the candidate points can be selected according to a setdisplacement therebetween or the image can be processed using (e.g.) anexhaustive search technique. Thus, as used herein the reference to theRANSAC technique should be taken broadly to include a variety of similarpoint-fitting techniques.

Additional functionalities of this system and method can be provided.These include support for mixed-polarity, automatically computing theprojection region width, support multi-view line-finding, and allowingthe input image be free of pre-warpage to remove optical distortion.These functionalities are described further below.

With further reference to the examples of FIGS. 14-16, the line-findingsystem and method of the illustrative embodiment generally supportsstandard LightToDark, DarkToLight and Either polarity settings(respectively) for the contrast between edges that are found. Inaddition, the system and method can also support a mixed-polaritysetting (FIG. 17) in which both a Light-To-Dark and Dark-to-Lightcharacteristic appears in the same line. Line-finding results of allfour settings are illustrated in the following figure. In anillustrative embodiment, the system and method can include amixed-polarity setting that allows finding of a single line thatcontains edge points of opposite polarities. This differs from aconventional setting of “Either” polarity in which all edge points of asingle line are either polarity—but only one polarity. Themixed-polarity setting can be advantageous when used to analyze thelight and dark checkerboards of (e.g.) a calibration plate, among otherapplications.

The user can select improved shift invariance of line-finding. In suchcase, the edge point extractor employs substantially overlappedprojection regions to improve result stability. When the regions arenon-overlapping, pixels under consideration can potentially move out ofthe projection regions when the image is shifted, resulting in poorshift invariance in line-finding results. Overlapped projection regionsensure that the pixels under consideration are continuously covered byprojection regions. If overlapped projection regions are used, thenincremental computation can be performed to maintain speed, along withpossible low-level optimization.

The user can provide masks that omit certain portions of the acquiredimage and/or imaged surface from analysis for line features. This can bedesirable where the surface includes known line features that are not ofinterest (e.g. barcodes that are analyzed by other mechanisms, text, andany other structures that are not germane to the task for which linesare to be found. Thus, the edge point extractor can support imagemasking where “don't care” regions in an image can be masked out, and“care” regions are masked in. Where such masking occurs, the coveragescores of the found lines is illustratively reweighted according to thenumber of edge points falling within the mask.

Reference is made to the exemplary image region 1800 of FIG. 18, whichshows coverage scores when image masks are present and the effect ofimage masking on such coverage scores. The edge point extractor supportsimage masking where “don't care regions” in an image can be masked out.As shown, the found line 1810 is characterized (based upon the “care”mask regions 1820) by care edge points. Such care edge points consist ofcare edge point inliers 1830 to the line 1810 and care edge pointoutliers 1840 to the line 1810. Don't care edge points 1850 on the line1810 reside between care regions 1820 of the mask, as shown in thisexample, and are not included in the coverage score computation, even ifthey reside on the line as inliers. Potential locations 1860 for edgepoints along the line 1810 are also determined, as shown. Thesepotential locations are positioned between known points at predictablespacing based on the spacing of found points. Illustratively, thecoverage scores of the found lines are reweighted according to thenumber of edge points falling within the mask. The coverage score is,thus, modified as follows:coverage score=number of care edge point inliers to line/(number of careedge point inliers to line+care edge point outliers to line+number ofcare potential locations of edge points).

After running the line-finding process according to the system andmethod herein, the found lines can be sorted in various ways based onsort criteria specified by the user (via (e.g.) the GUI). The user canchoose from intrinsic sort measures such as inlier coverage score,intensity or contrast. The user can also choose from extrinsic sortmeasures such as signed distance or relative angle. When using extrinsicsort measures, the user can specify a reference line segment againstwhich the extrinsic measures of the found lines are to be computed.

As described generally above, this system and method can include aMulti-Field-of-View (MFOV) overload, where a vector of images fromdifferent fields of view can be passed into the process. The imagesshould all be in a common client coordinate space based upon acalibration. As noted above, this functionality can be extremely helpfulin application scenarios where multiple cameras are used to capturepartial areas of a single part. Because the edge points retain gradientinformation, line features that are projected between gaps in the fieldof view can still be resolved (when the gradients in both FOVs match fora given line orientation and alignment in each FOV.

Notably, the system and method does not require (allows the image to befree-of) removal of warpage (i.e. does not require the image to beunwarped) to remove nonlinear distortion, assuming the distortion isnon-severe. Where the image is not unwarped, the system and method canstill detect candidate edge points, and map the point positions andgradient vectors through a nonlinear transform.

III. Line Labelling Training Interface and Runtime Process

With reference again to FIG. 1, the vision system process(or) 130further includes a label interface and associated process(or) 136 usedin training and runtime as described below. Additionally, the visionsystem process(or) includes, or interfaces with, a neural net, astatistical classifier and/or a K-nearest neighbor (K-NN classifier)process(or) 138, which receives image data and associated classifiersfrom a process(or) 137, which interfaces with the label process 136, asdescribed below. The label interface process(or) 136 operates attraining time to allow a user to associate specific (typicallytextual/alphanumeric) descriptions (herein termed “labels” or “tags”),with lines of interest in an image of an object. These processes(ors)enhance the functionality of the above-described line-finding tool byproviding the added ability to automatically label found lines in thetool results.

Reference is made to FIG. 19, which shows a procedure 1900 for traininga line-finding process, such as that described above. The trainingprocedure 1900 includes user-provided labels (typically in textualand/or alphanumeric form) for lines of interest. In step 1910, the userreviews a training image of the object—which is typically an actualimage of a model object, but can also be partially or fully, generatedby a CAD or other synthetic approach. The user identifies lines withinthe image that are of interest—for example the edge of a cover glass ina tablet or smart phone. Labels such as, “Inner Housing Inner Edge”,“Outer Housing Inner Edge”, etc. can be created to describe the lines ofinterest. The user accesses (from a list of possible labels) or createsa series of terms that define the lines and these are stored in adatabase for use in training the image (step 1920).

The user acquires or accesses one or more training images of the objectunder inspection by the vision system, and operates the trainingprocess(or) thereon (step 1930). The training process includes operationof the line-finding tool described above. The tool uses user-setparameters to automatically find multiple lines in each training image.

With reference also to FIG. 20, a user interface display screen 2000 isshown. The display screen 2000 contains a window 2010 that depicts atraining image of the object under inspection by the vision system. Theimage contains a series of found lines, generally denoted by highlightedindicia 2020, 2022, 2024, 2026, 2028 and 2029. The display highlights(often in a different color) of a particular line of interest 2032—theexemplary “Inner Housing Inner Edge” as described below that has beenclicked-upon or flagged by the user, to apply a label. Indicia 2022-2026relate to lines that are not of particular interest/relevance to thevision system task and/or have not been clicked upon. In general, a“line” as defined herein is essentially a linear step edge in an image,in which the line-finder returns a mathematical line fit as a result. Inthe case of the element 2030 (which is also a relevant edge to the taskand will be labelled by the user during training), two associated stepedges (see 2028 and 2029) are depicted on either side thereof.

The user, accesses a menu 2040 that includes the defined labels 2042,2044, 2046 and 2048. The user can click on the desired label using thecursor 2050 or other interface component, and then clicks on the foundline of interest (line 2030) to establish a label on that particularline (step 1940). Note that it is unnecessary to label all found lines,but only the relevant lines as desired by the user. If one or more ofthe relevant lines is/are missing from an image, then the labelassociated with that line remains unassigned.

After labeling the line results of the set of training images, the toolis trained and that data is stored in an appropriate database withrespect to the object/vision system task (step 1950). Subsequently, whenthe trained tool is run on images, the tool will not only find multiplelines, but it will also automatically assign to each found line a uniquelabel (or possibly no label if a found line is not relevant to theuser's application). This saves the user from having to post-processline results (e.g. in script code) to determine the identity of eachfound line.

According to (optional) step 1960, the identified (labelled) lines canbe provided to a neural network tool for processing and scoring of linefeatures of the image during runtime. Parameters are provided to theneural network outside of the training interface provided to the user,and can be pre-programmed—for example optimized parameters to search forline features in the image. Hence, the user is only responsible forassigning text labels to the line-finder results at train time. Theneural network (or another process—described generally below) is used toprobability-score candidate line features that are returned by the linefinding tool described above. More particularly, at training time, oncethe lines are found a training window 2010, and the user selects a namefor the lines he/she wants to label, a neural network classifier is alsocreated under for each of these labelled lines. The classifier canemploy the same (or similar) name as the labels that the user definedand applied in the training interface 2000. For example, if the userselects a line and assigns the label “Inner Housing Inner Edge” then theprocess creates a classifier with the same name, and adds the currentimage along with the line feature vector to the classifier.

A variety of commercially available neural network tools can beemployed, with appropriate programming that is customized or inaccordance with skill in the art, to extract line feature candidatesfrom in input image. It should also be clear that the above-describedline finding process is exemplary of a variety of line-finding tools andtechniques that deliver found lines from an image.

Reference is now made to the runtime procedure 2100 of FIG. 21 in whicha runtime image of the object is acquired and/or provided from aprevious acquisition process (step 2110). This image is passed to thevision system and associated processor 130, which operates the linefinding tool 134 described above based on the trained model, includingthe relevant labels (step 2120). The line-finding tool returns all foundlines, both labeled and unlabeled. Next, in step 2130, the neuralnetworks (or other processes using classifiers—described below) createdat train time are run, and a probability score map is obtained. Theprobability score map is a map of whether a pixel corresponds to thefeature vector for which the neural network has been trained. Thisprobability score map is the same size as image. Then each found line issampled at regular intervals, and the probability score is integratedfor each tool from the score map. In step 2140, labels are then assignedto lines based on which line has maximum probability score for eachlabel. In step 2150, the results of the line-finding step, withassociated label and probability score is stored and displayed for theuser, and/or employed for other downstream utilization tasks, such aspart inspection (pass/fail), robot control, etc.

FIG. 22 shows a display of runtime results on an exemplary runtimeobject (based upon the training object of FIG. 20) in which a relevantline (step edge) 2210 is highlighted. Additional relevant lines (e.g.lower highlight 2228 around linear feature 2310) are depicted, andcorrespond to other lines of interest that were trained in the trainingstep. Several non-relevant lines are also highlighted (2220, 2222,2224). The user has clicked upon one of the non-relevant lines(highlight 2220), which depicts “No Tag” in its respective informationwindow 2230, and shows a corresponding probability score of zero.Conversely, in FIG. 23, the display 2300, which shows the same resultsas display 2200 in FIG. 22, provides an information box 2320 for arelevant line 2210 with the label “Inner Housing Inner Edge”. Thisrepresents the user-labeled line feature from training time and theprobability score for such is 0.635, meaning that the found line is morelikely than not, the correct labeled one.

By way of example, the neural net classifier described hereinabovereceives an image (pixel data) as an input along with the featuresdefining the line segment. The output of the neural net classifier is aset of images where each pixel in a single image is the confidencewhether that corresponding input pixel coincides with the trained linesegment. The number of output images is the same as the number of linesegments the classifier has been trained upon. The desired output imagesthat the network is trained to reproduce on can be binary or grayscalerepresentations of the spatial probability distribution, narrow ridgesof high probability corresponding to the high gradient edges of the lineor other trained pattern. At runtime, the classifier receives an inputimage and generates a set of output images highlighting the areas wherethe neural net concludes that the trained line segment might beassociated with the current label/tag.

Alternatively, a classifier can be statistically trained. The inputs tothis statistically trained classifier can be provided as a featurevector that comprises measured properties of the current line segment(e.g. polarity, position, angle, etc.) along with measured propertiesthat describe the relationship between the current line segment and itsneighboring line segments (e.g. distance to the closest line, relativeangle, etc.) or computed properties of the image in the vicinity of theline segment (e.g. a 1D intensity image projection tangent to the linesegment, intensity histogram statistics, etc.). Accordingly, as usedherein, the term “classifier” can refer to a neural net classifier or astatistically trained classifier that produces labels. The term can alsorefer to a K-nearest neighbor (K-NN) classifier and/or process(or).Where a statistical classifier and/or K-NN classifier is employed, theoutput of probably scores or maps may be omitted from the procedures1900 and 2100, and not provided as part of the label/tag display in theinterface. However, such classifiers still advantageously allow forrefinement of the label process.

IV. Conclusion

It should be clear that the line-finder provided according to thesystem, and method and various alternate embodiments/improvements is aneffective and robust tool for determining multiple line features under avariety of conditions. In general, when used to find line features, thesystem and method has no particular limit on the maximum number of linesto be found in an image. Found lines can be labeled and classified sothat their probable correctness can be determined, thereby increasingthe versatility and robustness of the line-finding process.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention.Features of each of the various embodiments described above may becombined with features of other described embodiments as appropriate inorder to provide a multiplicity of feature combinations in associatednew embodiments. Furthermore, while the foregoing describes a number ofseparate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. For example, asused herein the terms “process” and/or “processor” should be takenbroadly to include a variety of electronic hardware and/or softwarebased functions and components (and can alternatively be termedfunctional “modules” or “elements”). Moreover, a depicted process orprocessor can be combined with other processes and/or processors ordivided into various sub-processes or processors. Such sub-processesand/or sub-processors can be variously combined according to embodimentsherein. Likewise, it is expressly contemplated that any function,process and/or processor herein can be implemented using electronichardware, software consisting of a non-transitory computer-readablemedium of program instructions, or a combination of hardware andsoftware. Additionally, as used herein various directional anddispositional terms such as “vertical”, “horizontal”, “up”, “down”,“bottom”, “top”, “side”, “front”, “rear”, “left”, “right”, and the like,are used only as relative conventions and not as absolutedirections/dispositions with respect to a fixed coordinate space, suchas the acting direction of gravity. Additionally, where the term“substantially” or “approximately” is employed with respect to a givenmeasurement, value or characteristic, it refers to a quantity that iswithin a normal operating range to achieve desired results, but thatincludes some variability due to inherent inaccuracy and error withinthe allowed tolerances of the system (e.g. 1-5 percent). Accordingly,this description is meant to be taken only by way of example, and not tootherwise limit the scope of this invention.

What is claimed is:
 1. A system for finding line features in an acquiredimage based upon one or more cameras comprising: a vision systemprocessor; an interface associated with the vision system processor,that allows creation of discrete labels with respect to relevant lineslocated by a line-finding process in a training image of the object; aruntime line-finding process that locates lines in an acquired image; aneural net process that employs classifiers, based on the labels, todetermine a probability map for line features relative to the labels;and a runtime result-generation process that provides labels andprobability scores for at least one of the relevant lines.
 2. The visionsystem as set forth in claim 1 wherein the runtime result-generationprocess provides probability scores for non-relevant lines.
 3. Thevision system as set forth in claim 1 wherein the result-generationprocess includes an interface that highlights lines and provides theprobability scores associated with the highlighted lines.
 4. The visionsystem as set forth in claim 1 wherein the probably score map is similarin size to the acquired image.
 5. The vision system as set forth inclaim 1 wherein the classifier process employs at least one of a neuralnet classifier and a statistically trained classifier.
 6. The visionsystem as set forth in claim 1 wherein the line-finding processcomprises a processor that receives image data of a scene containingline features, having an edge point extractor that (a) computes agradient vector field from the image data, (b) projects the gradientvector field over a plurality of gradient projection sub-regions, and(c) finds a plurality of edge points in respective of the gradientprojection sub-regions based on the projected gradient data; and aline-finder that generates a plurality of lines that are consistent withthe edge points extracted from the image.
 7. The system as set forth inclaim 6 wherein the line-finder operates a RANSAC-based process to fitinlier edge points to new lines including iteratively defining linesfrom outlier edge points with respect to previously defined lines. 8.The system as set forth in claim 6 wherein the gradient field projectionis oriented along a direction set in response to an expected orientationof one or more or the line features.
 9. The system as set forth in claim6 wherein the gradient field projection defines a granularity based on aGaussian kernel.
 10. The system as set forth in claim 6 wherein the edgepoint extractor is arranged to find a plurality of gradient magnitudemaxima in each of the gradient projection sub-regions, wherein thegradient magnitude maxima are respectively identified as some of theplurality edge points, being described by a position vector and agradient vector.
 11. The system as set forth in claim 6 wherein theline-finder is arranged to determine consistency between at least oneedge point of the extracted plurality of edge points and at least onecandidate line of the found plurality of lines by computing a metricthat is based upon a distance of the at least one edge point from the atleast one candidate line and an angle difference between a gradientdirection of the at least one edge point and a normal direction of theat least one candidate line.
 12. A system for finding line features inan acquired image based upon one or more cameras comprising: a visionsystem processor; an interface associated with the vision systemprocessor, that allows creation of discrete labels with respect torelevant lines located by a line-finding process in a training image ofthe object; a runtime line-finding process that locates lines in anacquired image; and a statistical classifier that produces the labelsfor the interface based upon lines located by the line-finding process.13. A system for finding line features in an acquired image based uponone or more cameras comprising: a vision system processor; an interfaceassociated with the vision system processor, that allows creation ofdiscrete labels with respect to relevant lines located by a line-findingprocess in a training image of the object; a runtime line-findingprocess that locates lines in an acquired image; and a K-NN classifierthat produces the labels for the interface based upon lines located bythe line-finding process.
 14. A method for finding line features in anacquired image, based upon one or more cameras, comprising the steps of:providing an interface associated with the vision system processor thatallows creation of discrete labels with respect to relevant lineslocated by a line-finding process in a training image of the object;locating, with a runtime line-finding process, found lines in anacquired image; and generating, with a classifier, labels for at leastone of the relevant found lines.
 15. The method as set forth in claim 14wherein the classifiers include at least one neural net classifier, andfurther comprising, employing the at least one neural net classifierthat, based on the labels, determine a probability map for found linefeatures relative to the labels, and generating provides probabilityscores for non-relevant found lines.
 16. The method as set forth inclaim 15 wherein the step of generating includes highlighting, in aninterface, found lines and providing the probability scores associatedwith the highlighted lines.
 17. The method as set forth in claim 14wherein the classifiers are at least one of neural net classifiers,statistically trained classifiers and K-NN classifiers.
 18. The methodas set forth in claim 14 wherein the line-finding process receives imagedata of a scene containing line features, having an edge point extractorthat (a) computes a gradient vector field from the image data, (b)projects the gradient vector field over a plurality of gradientprojection sub-regions, and (c) finds a plurality of edge points inrespective of the gradient projection sub-regions based on the projectedgradient data; and computing a plurality of lines that are consistentwith the edge points extracted from the image.
 19. The method as setforth in claim 18 wherein the step of computing operates a RANSAC-basedprocess to fit inlier edge points to new lines including iterativelydefining lines from outlier edge points with respect to previouslydefined lines.
 20. The method a set forth in claim 18 wherein thegradient field projection is oriented along a direction set in responseto an expected orientation of one or more or the line features.
 21. Themethod as set forth in claim 18 wherein the edge point extractor finds aplurality of gradient magnitude maxima in each of the gradientprojection sub-regions, wherein the gradient magnitude maxima arerespectively identified as some of the plurality edge points, beingdescribed by a position vector and a gradient vector.
 22. The method asset forth in claim 18 wherein the line-finder determines consistencybetween at least one edge point of the extracted plurality of edgepoints and at least one candidate line of the found plurality of linesby computing a metric that is based upon a distance of the at least oneedge point from the at least one candidate line and an angle differencebetween a gradient direction of the at least one edge point and a normaldirection of the at least one candidate line.